1 Speech Is at Least 4 - Dimensional : Receptive Fields in Time - Frequency

The successful integration of temporal information is crucial for speech recognition: for example, dynamic time warping and more recently hidden Markov models (HMMs) have been critical in speech recognition technology, and both these methods amount to time-aligning feature vectors to stored word and sentence templates. The correlogram [14] relies on the temporal processing of individual outputs of a cochlea-inspired lter bank. Furui [6] posited the existence of perceptual critical points located in time and containing information crucial for the perception of speech, and this idea inspired the statistical speech recognition model SPAM [11] which focusses modeling power on points of transition or maximal spectral change. This suggests that certain time points of a speech signal are in some sense more \important" than others for a recognition task. With HMMs, the locations of the state transitions in uence the particular realization of a model and therefore its likelihood; states between transitions are of secondary importance. The SPAM results [1, 12] in fact show that we can group these \nontransitional" states together into a single broad category and still achieve a good recognition error rate. All the above methods, however, neglect another potential axis of discrimination, namely the cross-spectrotemporal co-information. Non-convex or disjoint patterns with signi cant temporal and spectral extent cannot be detected directly. Instead, existing methods focus either across frequency on a small slice of time (e.g., LPC cepstral features of a single speech frame) or across time on a small slice of frequency (e.g., the correlogram). There are in fact several results suggesting that the utilization of cross spectro-temporal co-information can have a bene cial e ect on speech processing and recognition. In the arti cial neural network (ANN) speech community [10] it has been shown that using multi-frame context windows can improve recognition scores. The loss of independent feature vectors notwithstanding, ANNs with such \wide" context windows have the potential to learn timeand frequencylike patterns, depending on the features used. In [2], it was shown that a cross-channel correlation algorithm can be used to nd formants in voiced speech in high noise situations. In [5], it was shown that using cross-channel correlation can be used to identify individual sound sources in a mixed auditory scene. Also, in [8], it was suggested that using long-term cross-channel correlation could be used as a measure of speech quality. It can also be argued that the use of cross-spectrotemporal information is biologically plausible. Echoic memory is a temporary bu er in the auditory system that holds pre-attentive information for a brief period of time before subsequent, more detailed, and more taxing processing takes place [13]. It is likely that this storage occurs at the post-cochlear level as we have no evidence for such memory before or during cochlear processing. Therefore, the echoic storage can plausibly be thought of as a form of processed spectro-temporal bu er. Thus assumed, it would be surprising if subsequent processing did not attempt to nd patterns utilizing not just the temporal or spectral axes alone, but shaped regions spanning both time and frequency. Therefore, it may be postulated that the auditory system has the capability, over a 200ms time-span comparable to echoic store, to observe the co-occurrence of information in di erent spectro-temporal regions. Similar to the cells in the visual system that respond to particular shapes, one may consider receptive elds over a form of post-cochlear spectro-temporal plane. Later stages of the auditory system could derive arbitrarily shaped regions that perhaps dynamically scale, shift, and transform according to a variety of control mechanisms. In this paper, we consider a new representation of speech that attempts to explicitly represent non-convex spectrotemporal co-information. Section 2 discusses the computational aspects of our representation. Section 3 illustrates with an example. And nally, section 4 discusses current and future work.